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SUMMARY 


Managing  a  test  and  evaluation  (T4E)  of  a  devefopment  system  is  a  cnaiienge  Every 
system  is  unique  and  has  different  requirements  However  tfvere  are  certain  giooai  issues  that 
pertain  to  each  and  every  T&E  Among  these  are  defining  oOjectives  seeping  me  T&E  as 
simple  as  possible,  creating  a  dynamic  test  ptan  and  ensuring  open  communication  oetween 
the  involved  personnel  While  these  items  do  f>oi  ensure  a  successful  test  ignoring  mem  wiii 
almost  guarantee  a  failure 
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A  TEST  MANAGER'S  OPERATIONAL  GUIDE 


I.  INTRODUCTION 

For  over  10  years  the  Air  Force  has  recognized  deficiencies  in  the  existing  enlisted  on-the-job 
training  (OJT)  system.  The  Air  Force  Human  Resources  Laboratory  (AFHRL)  conducted  a  study 
of  the  OJT  system  in  1975.  and  the  Air  Force  Inspector  General  (IG)  performed  a  functional 
management  inspection  (FMl)  of  OJT  in  1977.  Both  efforts  produced  similar  results:  the  OJT 
system  was  seriously  deficient  in  training  definition,  delivery,  evaluation,  administration,  and 
personnel  utilization.  As  part  of  a  series  of  Air  Staff-directed  initiatives  designed  to  correct 
these  deficiencies,  a  follow-on  study  outlined  the  functional  and  automation  requirements  for  Air 
Force  OJT,  developed  a  syste.n  specification,  and  picked  a  site  for  the  design,  development, 
test,  and  evaluation  of  an  Advanced  On-the-job  Training  System  (AOTS). 

Beginning  in  1985,  the  team  of  AFHRL.  Douglas  Aircraft  Company  (DAC),  and  Ball  Systems 
Engineering  Division  (BSED)  began  designing  the  prototype  AOTS  at  Bergstrom  AFB,  Texas.  A 
system  level  test  and  evaluation  (SLT&E)  followed  the  successful  completion  of  the  development 
work  in  August  of  1988.  The  SLT&E  lasted  for  1  year;  results  indicated  that  the  work  center 
users  of  the  AOTS  liked  it,  thought  it  was  easy  to  use,  felt  it  solved  the  deficiencies  of  the 
existing  OJT  system  and  could  be  effectively  used  in  the  operational  environment,  and  felt  it 
would  improve  the  performance  of  individual  trainees  and  the  whole  OJT  system  over  time. 
The  test  was  a  success. 

But  a  successful  T&E  does  not  just  happen,  and  a  T&E  will  not  run  itself.  There  are 
always  too  many  people  to  lead,  too  many  measures  or  instruments  to  create,  and  too  many 
last-minute  schedule  crises  to  overcome.  A  test  manager  is  needed  to  guide  the  effort  so  that 
each  system  is  evaluated  properly,  results  are  produced,  and  the  T&E  is  conducted  within  time 
and  dollar  constraints.  Based  on  the  experience  gained  during  the  AOTS  SLT&E,  a  test  manager 
should  use  the  following  principles  as  a  basic  recipe  for  the  successful  planning  and  execution 
of  a  T&E. 


11.  OBJECTIVE 

In  the  test  and  evaluation  (T&E)  of  any  system,  the  most  important  step  is  defining  the  T&E 
objective.  There  may  be  multiple  objectives,  but  these  should  be  kept  to  a  manageable  number 
(four  or  less).  The  act  of  unambiguously  defining  the  overall  test  objectives  will  focus  the 
efforts  of  all  those  involved,  acting  as  unifying  principles  throughout  the  test.  Muddled,  vague, 
or  poorly  defined  objectives  will  result  in  similar  test  results  and  conclusions. 

A  critical  prerequisite  to  defining  test  objectives  is  learning  as  much  as  possible  about  the 
system  to  be  tested.  Since  T&E  strategy  definition  begins  early  in  the  development  of  a  system, 
this  is  necessarily  an  ongoing  process.  Only  when  the  system  is  fully  understood  can  precise, 
quantifiable  measurement  and  testing  of  its  capabilities  be  planned,  designed,  and  performed. 

An  important  distinction  to  make  is  that  between  the  testing  of  the  functions  and  capabilities 
of  a  system  versus  the  evaluation  of  the  effect  the  system  has  upon  the  users  and  their 
environment  (physical  environment,  procedures,  job  or  time  requirements,  support,  etc.).  Both 
aspects  are  important  in  testing  a  system,  but  confusing  the  two  during  the  planning  and 
evaluation  processes  will  produce  data  that  is  difficult  to  distill  and  analyze  meaningfully. 

Teamwork  and  team  building  are  essential.  Try  to  get  people  on  the  T&E  team  that  want 
to  be  on  it.  Air  Force,  contractor,  user,  and  all  other  involved  personnel  must  be  active 
participants  in  defining  and  achieving  the  objectives  of  the  test  program.  When  all  parties  are 
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involved  in  the  planning  phase,  the  test  becomes  theirs,  and  interest  in  successfully  completing 
the  test  will  be  greater 


III.  GUIDANCE 

Seek  guidance  from  above.  Regulations  governing  T&E  (APR  80-14.  AFTECR  55-i.  DOD 
Directive  5000.3,  etc.)  cannot  and  should  not  be  ignored.  Most  T&Es  have  requirements 
commensurate  with  the  stage  of  development  of  the  system.  Regulations,  binding  or  not.  often 
contain  excellent  suggestions  and  methods  for  planning  and  conducting  a  T&E  and  form  a 
good  foundation  for  the  T&E  program.  Look  for  lessons  learned  from  other  similar  programs, 
avoid  past  mistakes  and  improve  future  test  performance  and  results. 

Many  systems  will  also  have  political,  cost,  or  performance  constraints  and  goals  attached 
These  guides  and  constraints,  such  as  the  Program  Management  Directive  (  PMD).  Statement 
of  Need  (SON),  or  Statement  of  Work  (SOW),  must  be- considered  an  integral  part  of  the  T&E 
development  process.  Priority  should  be  given  to  "answering  the  mail,"  especially  if  the  constraints 
and  goals  are  imposed  by  the  future  users  of  the  system. 


IV.  TEST  PLANNING 

The  Keep  It  Simple,  Stupid  (KISS)  principle  should  permeate  the  entire  T&E.  Obviously. 
T&E  requirements  must  be  satisfied  with  the  appropriate  level  of  complexity-you  cannot  measure 
a  hypervelocity  weapon  with  a  wristwatch.  However,  a  test  objective  hierarchy  should  be 
formulated,  starting  with  the  basic,  fundamental  questions  (Does  it  work?;  Do  the  users  like  it?; 
etc.)  and  branch!  ng  outward  and  downward.  The  fundamental  questions  should  have  priority 
in  the  planning  and  execution  of  the  T&E.  Only  when  the  basic  questions  are  addressed  should 
you  move  on  to  greater  levels  of  detail  (critical  issues,  critical  questions,  specific  measures, 
etc.). 

After  test  objectives  are  defined  (again,  an  evolutionary  process),  the  next  step  is  to  perform 
long-ranged  planning  steps.  The  first  of  these  steps  is  to  create  a  T&E-specific  work  breakdown 
structure  (WBS)  to  organize  at  a  high  level  the  work  to  be  done  before,  during,  and  after  the 
test.  The  WBS  should  be  defined  down  to  a  level  of  detail  where  the  lowest  category  requires 
no  more  than  15  or  so  discrete  tasks  to  complete  it.  Refer  to  Appendix  A  for  the  WBS  used 
on  the  AOTS  project. 

These  discrete  tasks  should  be  defined  for  each  final  WBS  unit;  specificity  is  the  rule  here. 
Each  task  should  have  a  definite  beginning  and  end,  and  be  the  responsibility  of  one  person. 
One  way  to  organize  the  tasks  is  to  devote  a  separate  sheet  of  paper  to  each  final-level  WBS 
element.  The  tasks  needed  to  accomplish  each  element  are  then  listed  on  that  sheet,  along 
with  the  person  responsible  for  the  task,  duration  of  the  task,  resources  necessary  to  perform 
the  task,  any  time  con  straints,  and  other  relevant  information. 

The  next  step  is  to  network  the  tasks.  The  networking  process  involves  assigning  responsibility 
for  each  of  the  tasks  to  a  single  point  of  contact  (POC)  and  deciding  which  tasks  should 
precede  the  others.  A  white  board  and  either  Post-lt'^'^^  notes  or  paper  and  tape  are  ide  al 
for  this  sequencing  exercise.  After  assembling  the  prototype  network,  a  list  of  tasks  and  their 
succes  sors  should  be  created.  Loading  the  task  data  and  the  predecessor-successor  lists  into 
a  network  analysis  and  pi  anning  software  package  such  as  the  Primavera  Project  Planner*^^* 
(P3)  or  the  Computer-Supported  Network  Analys  is  System  (CSNAS)  is  a  good  idea.  Large 
T&E  efforts  are  easier  to  manage  using  the  Program  Evaluation  Review  T  echnique  (PERT) 
charting,  critical  path  analysis,  resource  tracking,  and  follow-up  capabilities  of  packages  such 
as  these. 
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First,  check  the  predecessor-successor  logic  of  your  network  by  producing  a  logic  diagram 
unconstrained  by  dates.  The  next  product  should  be  a  time-constrained  logic  diagram,  showing 
all  the  tasks,  relationships  desired,  and  any  absolute  "drop-dead"  dates.  Careful  study  of  the 
se  products  will  give  managers  a  good  idea  whether  or  not  the  plan,  as  designed,  will  meet 
schedule  constraints  A  tool  such  as  this  is  ideal  for  playing  "what  if?  "  scenarios  and  determining 
the  best  sequence  of  events.  Always  remember  that  these  are  merely  tools:  they  will  not  make 
decisions  and  are  only  as  good  as  the  information  loaded  in  them. 

At  this  point,  the  test  manager  has  objectives,  critical  issues,  sub-objectives,  networks,  and 
charts,  to  name  but  a  few  things.  These  are  the  heart  of  the  T&E  master  plan.  The  main 
point  to  remember  about  the  plan  is  that  it  will  constantly  change,  and  that  change  needs  to 
be  controlled.  One  effective  way  of  controlling  the  change  is  to  segment  the  plan  into  logical 
sections,  each  with  its  own  page  number  sequence.  Thus,  each  change  might  only  affect  a 
small  section,  not  a  whole  test  plan.  Approved  changes  to  the  plan  should  be  registered  in 
pen  in  the  test  manager"s  copy.  This  is  the  master  copy  of  the  plan.  Change  sheets  can  then 
be  issued  periodically  to  others  having  copies  of  the  plan. 


V.  TEST  EXECUTION 

The  project  planning  and  analysis  tools  are  also  helpful  in  the  follow-up"  aspect  of  test 
execution.  Task  listings,  ordered  by  POC  and  scheduled  dates,  are  extremely  helpful  to  the 
individuals  responsible  as  reminders  of  when  tasks  need  to  be  finished,  thus  establishing  peri¬ 
odic,  achievable  goals  for  those  individuals.  They  are  also  good  for  the  manager  as  monthly 
"to  do"  lists  and  reminders  to  communicate  directly  with  the  POCs.  Other  products  such  as 
PERT  charts,  GANTT  charts,  etc.  are  helpful  in  that  they  keep  individual  POCs  aware  of  their 
role  in  the  overall  T&E  team  effort. 

As  the  test  is  being  executed,  periodic  get-togethers  of  the  involved  parties  (Air  Force, 
contractor,  user)  will  help  the  team  members  remain  committed,  increase  communication,  and 
improve  the  overall  effort.  One  successful  strategy  was  having  the  main  T&E  representatives 
from  each  group  (Air  Force,  contractor)  meet  each  Monday  morning  to  discuss  the  upcoming 
week’s  activities  and  to  address  unresolved  issues.  This  small  group  (two-three  people)  would 
solve  those  problems  they  could  solve  personally,  delegate  or  team  up  on  solvable  problems 
requiring  more  expertise,  and  defer  to  the  weekly  test  working  group  those  problems  requiring 
higher  level  (program  manager)  involvement.  An  important  point  is  not  to  rely  on  meetings  to 
discuss  and  solve  problems.  Foster  initiative  by  delegating  and  creating  teams  to  solve  problems. 
And  limit  the  size  of  meetings,  unless  they  are  strictly  informational.  A  large  group  discussing 
problems  usually  gets  very  little  done. 

No  matter  what  the  item  to  be  tested,  be  it  a  missile  or  an  automated  training  system,  you 
must  keep  a  running  account  of  the  context  in  which  and  the  process  by  which  the  test  was 
executed.  These  include  the  physical  and  regulatory  environments,  internal  and  external  change 
s  to  the  project  and  the  test,  human  factors  information  regarding  the  participants,  and  descriptions 
of  how  the  test  played  itself  out,  to  list  a  few.  These  data  will  be  essential  in  interpreting  the 
test  and  analysis  results;  often  times  meaningful  conclusions  are  impossible  without  these  context 
and  process  data. 


VI.  CONCLUSION 

The  preceding  recommendations  are  not  intended  to  be  an  all-inclusive  cookbook  for  a 
perfect  T&E.  The  main  points  of  objectives,  simpiicity,  olanning,  and  control  will  help  any  future 
manager  get  the  T&E  started  correctly  and  will  guide  the  test  team  to  a  successful  conclusion. 
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